Cloudflare Launches New Web Scraping Management Regulations, Effectively Blocking AI Training-Related Web Scrapers Accessing Ad Pages Starting September 15
Cloudflare updated its web scraping rules on July 1st, marking behaviors such as search, AI proxy, and training. It is scheduled to effectively block AI-related web scrapers from accessing pages with advertisements starting September 15, 2026. Webmasters may wish to be indexed but need to protect their copyright revenue, while a large number of mixed web scrapers are difficult to distinguish. Its manager will identify the full purpose of all web scrapers, and if any prohibited attribute is detected, it will trigger a block.